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EXECUTIVE  SUMMARY 

This  report  documents  the  fiscal  year  (FY)  2000  results  of  the  Federal  Aviation 
Administration  (FAA)  Aircraft  Certification  Service  (AIR)  Aircraft  Certification  Systems 
Evaluation  Program  (ACSEP). 

The  ACSEP  was  designed  to  determine  if  FAA  production  approval  holders  and 
delegated  facilities  are  complying  with  the  requirements  of  applicable  Code  of  Federal 
Regulations  (CFR)  and  the  procedures  established  to  meet  those  requirements.  It  also 
surveys  the  application  of  standardized  industry  practices,  not  required  by  the  CFR  or 
FAA-approved  data,  to  identify  national  trends  that  may  require  development  of  new  or 
revised  regulations,  policy,  or  guidance.  The  elements  of  the  evaluation  are  referred  to  as 
criteria.  Data  was  collected  on  noncompliance  and  applicability  with  respect  to  those 
criteria.  The  background  of  ACSEP,  a  program  overview,  the  process  for  scheduling 
evaluations,  and  training  evaluators  are  discussed  in  appendix  A. 

During  an  ACSEP  evaluation,  the  actual  operating  practices  of  a  facility  are  compared  to 
the  CFR,  FAA-approved  data,  and  the  facility’s  internal  procedures.  Any  inconsistency 
discovered  (termed  “issue”  in  this  report)  is  classified  and  recorded.  An  issue  is 
classified  by  its  type  and  the  system  element  under  which  it  is  noted.  There  are  five  issue 
types: 


Safety  Finding  -  an  issue  that  compromises  immediate  continued  operational 
safety. 

Systemic  Finding  -  an  issue  that  is  systemic  in  nature,  i.e.,  is  pervasive,  repeatable, 
or  represents  a  breakdown  in  the  quality  management  system.  For 
an  issue  to  be  categorized  a  finding,  it  must  also  be  a  noncompliance 
to  a  CFR  or  FAA-approved  data  (or  a  noncompliance  with  the 
procurement  instrument  when  a  facility  is  a  supplier). 

Systemic  Observation  -  an  issue  that  is  systemic  in  nature  and  is  a  noncompliance 
to  facility  procedures  that  are  not  FAA  approved. 

Isolated  Observation  -  an  issue  that  is  of  an  isolated  or  nonsystemic  nature,  i.e., 
isolated  to  a  particular  person  and/or  timefi’ame  and  does  not 
represent  a  breakdown  in  the  quality  management  system.  For  an 
issue  to  be  categorized  an  isolated  observation,  it  must  also  be  an 
isolated  noncompliance  to  a  CFR  or  FAA-approved  data  (or  a 
noncompliance  with  the  procurement  instrument  when  a  facility  is  a 
supplier). 

CFR-Based  Observation  -  the  discovery  of  FAA-approved  data  that  is 
inconsistent  with  the  CFR. 
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Issues  are  classified  using  system  elements.  In  total,  there  are  17  system  elements  that 
represent  a  quality  management  system  for  a  production  approval  holder: 


Organization  and  Responsibility 
Design  Data  Control 
Software  Quality  Assurance 
Manufacturing  Processes 
Special  Manufacturing  Processes 
Statistical  Quality  Control  (SQC) 
Tool  and  Gauge 
Testing 

Nondestructive  Inspection 


Supplier  Control 
Nonconforming  Material 
Material  Handling/Storage 
Airworthiness  Determination 
FAA  Reporting  Requirements 
Internal  Audit 
Global  Production 
Manufacturing  Maintenance  Facility 


There  are  10  system  elements  that  represent  a  quality  management  system  for  a  delegated 
facility: 


•  Organization  and  Responsibility  •  Project  Management 

•  Design  Data  Approval  •  Design  Change  Approval 

•  Testing  •  Conformity  Inspection 

•  Airworthiness  Certification  •  FAA  Notification 

•  Continued  Airworthiness  •  Audit 

Each  system  element  is  further  divided  into  “criteria.”  To  fully  examine  the  detailed 
areas  within  each  of  the  17  system  elements,  the  criteria  were  developed  with  extensive 
assistance  from  industry.  A  process  also  exists  to  identify  potential  new  criteria  should 
the  existing  criteria  not  address  a  particular  functional  area  within  a  system  element.  The 
subclassification  of  issues  into  detailed  criteria  allows  the  FAA  to  identify  specific  areas 
of  concern  and  allows  industry  to  focus  corrective  action  on  those  specific  areas  of 
concern.  For  example,  the  supplier  control  system  element  is  composed  of  16  individual 
criteria.  Specific  areas  of  concern  that  may  be  identified  include:  the  use  of  approved 
suppliers,  periodic  evaluations  of  suppliers,  flowdown  of  applicable  technical  and  quality 
requirements  to  suppliers,  raw  material  verification,  and  others. 

Analysis  Results  and  Conclusions 

Of  the  610  issues  recorded  at  the  291  facilities  evaluated  in  FY  2000,  one  identified  a 

sigmficant  safety  concern,  i.e.,  a  finding  for  which  immediate  corrective  action  was 
required. 

There  were  no  safety  findings,  recorded  for  a  PAH.  There  was  one  safety  finding 
recorded  at  a  delegated  facility  in  the  area  of  Design  Data  Approval  (specificcilly  criteria 
"Control  of  Type  Design  Data)  for  failure  to  include  required  safety  system 
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information  in  the  STC  data  package.  The  balance  of  the  issues  reported  were  not 
considered  an  immediate  safety  concern. 

The  system  elements  where  the  most  issues  were  reported  are  as  follows: 

Manufacturing  Processes  -  Specific  functions  and  operations  necessary  for  the 

fabrication  and  inspection  of  parts  and  assemblies  (e.g.,  machining,  riveting,  and 
assembling). 

Supplier  Control  -  The  system  by  which  the  evaluated  facility  ensures  that  supplier 
materials,  parts,  and  services  conform  to  FAA-approved  design. 

Design  Data  Control  -  The  planning  and  integration  of  the  evaluated  facility's  procedures 
for  continuously  maintaining  the  integrity  of  design  data,  as  approved  by  the 
FAA  or  FAA-delegated  representatives,  in  the  completed  product.  This  includes 
software  used  in  type-certificated  aircraft  or  related  products  (airborne  software). 

Tool  and  Gauge  -  The  function  which  establishes  control  of  precision  measuring  devices 
(e.g.,  tools,  scales,  gauges,  fixtures,  instruments,  or  automated  measuring 
machines)  used  in  fabrication,  special  processing,  inspection,  and  testing  of  detail 
parts,  assemblies,  and  completed  products  to  determine  conformity  to 
FAA-approved  design. 

Nonconforming  Material  -  The  method  of  controlling,  evaluating,  and  dispositioning  of 
any  part/product  which  does  not  conform  to  FAA-approved  design. 

Special  Manufacturing  Processes  -  The  methods  whereby  materials,  parts,  or  assemblies 
are  worked  or  fabricated  through  a  series  of  precisely  controlled  steps,  and  which 
xmdergo  physical,  chemical,  or  metallurgical  transformation  (e.g.,  heat-treating, 
brazing,  welding,  and  processing  of  composite  material). 

The  first  five  of  the  above  six  system  elements  have  been  the  most  predominant  areas  for 
issues  since  a  baseline  for  the  data  was  set  in  FY  1995.  A  more  detailed  discussion  of  the 
data  is  presented  throughout  Section  3  of  the  report. 

An  area  of  special  focus  is  chosen  aimually  to  determine  if  increased  oversight  is 
required.  Software  Quality  Assurance  was  chosen  as  the  special  focus  for  this  fiscal  year. 
There  were  a  total  of  six  systemic  findings,  two  systemic  observations,  three  isolated 
observations,  and  one  CFR-based  observation.  These  were  equally  distributed  amongst 
the  specific  criteria  within  this  system  element.  No  definitive  conclusions  can  be  drawn 
on  these  results  at  this  time.  It  is  expected  that  the  number  of  issues  recorded  in  this  area 
will  increase  over  time  based  on  industry’s  greater  reliance  on  software  driven 
technologies.  As  more  manufacturing  systems  develop  a  greater  dependence  on  software 
driven  systems,  this  area  will  require  greater  diligence  in  its  surveillance  by  PAH  quality 
assurance  and  audit  personnel.  These  personnel  will  have  to  ensure  that  they  remain  well 
versed  in  the  current  software  technologies  and  software  system  audit  techniques. 
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The  continuous  improvement  initiatives  implemented  in  ACSEP  have  resulted  in  a  steady 
increase  in  reported  favorable  experiences  by  evaluation  teams  during  ACSEP 
evaluations  over  the  last  six  years.  Evaluation  teams  in  FY  2000  reported  96  percent 
fewer  problems  in  interpreting  and  utilizing  the  ACSEP  order  and  performing  evaluations 
than  in  FY  1995.  In  addition,  there  have  been  continuous  improvements  in  customer 
satisfaction  with  ACSEP  evaluations.  As  part  of  the  ACSEP  continuous  improvement 
process,  the  facility’s  management  is  provided  with  a  feedback  report  on  which  to  record 
their  assessment  of  the  conduct  of  the  evaluation  team.  All  phases  of  an  ACSEP 
evaluation  are  addressed  from  pre-evaluation  notification  through  post-evaluation  review 
of  any  findings  and/or  observations.  Less  than  one  percent  of  the  facilities  returning  a 
feedback  report  in  the  last  three  years  have  reported  dissatisfaction  with  the  conduct  of 
the  ACSEP  evaluation  teams.  See  Section  4  for  additional  information  on  the  continuous 
improvement  program  of  ACSEP. 
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FY  2000  Report 

1.  Introduction 

This  report  summarizes  the  results  of  the  Aircraft  Certification  Systems  Evaluation 
Program  (ACSEP)  and  provides  a  comprehensive  view  of  the  program's  results  from 
October  1999  through  September  2000.  The  presentation  of  the  data  provides  insight  into 
procedural  compliance  trends  with  production  approval  holders. 

1.1  Report  Structure 

Section  1  provides  an  introduction  and  overview  of  the  program  status. 

Section  2  provides  a  summary  of  the  data  presented  in  this  report. 

Section  3  provides  a  consolidation  of  the  data  that  led  to  the  conclusions  presented  in 
Section  2. 

Section  4  provides  the  results  of  the  ACSEP  improvement  effort  including  feedback  from 
industry,  lessons  learned,  and  comments  received  regarding  the  ACSEP  evaluations. 

There  are  two  appendices:  Appendix  A  provides  a  brief  history  and  backgroxmd  of 
ACSEP  and  Appendix  B  provides  definitions.  Previous  ACSEP  Annual  Reports  included 
an  appendix  providing  detailed  data  tables  regarding  the  number  and  percentage  of 
occurrence  of  an  issue  for  each  specific  criteria.  This  information  will  now  be  provided 
on  the  AIR-200  web  page  and  may  also  be  requested  from  AIR-200  at  (202)  267-8361 . 
The  address  for  the  web  page  is  http://www.faa.gov/avr/air/air200/200home.htm. 
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1.2  Program  Overview  of  ACSEP 

This  subsection  provides  an  overview  of  the  ACSEP  and  a  brief  history  of  its  growth. 

The  ACSEP  was  developed  as  a  result  of  numerous  years  of  experience  with  Quality 
Assurance  Systems  Analysis  Review  (QASAR)  audits  and  observations  made  during  an 
interim  audit  program  called  “Operation  SNAPSHOT.” 

a)  ACSEP  evaluations  are  performed  in  accordance  with  consistent  and 
standardized  evaluation  criteria. 

b)  The  evaluation  criteria  used  during  an  ACSEP  evaluation  were  developed  with 
extensive  input  and  cooperation  from  the  aviation  industry  to  ensure  that 
emerging  technologies  were  addressed. 

c)  ACSEP  evaluation  results  are  maintained  in  a  centralized  database. 

d)  An  annual  report  of  the  aggregate  ACSEP  evaluation  results  is  published. 

e)  ACSEP  actively  incorporates  the  evaluation  of  facilities  with  engineering 
delegations.  The  facilities  that  are  evaluated  by  ACSEP  are: 

•  Approved  Production  Inspection  System  (APIS) 

•  Production  Certificate  (PC)  and  Production  Certificate  Extension 
(PCEX) 

•  Parts  Manufacturer  Approval  (PMA) 

•  Technical  Standard  Order  (TSO)  authorization 

•  Delegation  Option  Authorization  (DOA) 

•  Designated  Alteration  Station  (DAS) 

•  Special  Federal  Aviation  Regulation  No.  36  (SFAR-36) 

1.3  Significant  Events  During  the  Fiscal  Year 

The  following  significant  events  either  changed  policy  that  affects  the  structure  of 
ACSEP,  are  measures  intended  to  improve  PAH  quality  systems  thereby  reducing 
findings  and  observations,  or  are  significant  activities  initiated  as  a  result  of  ACSEP 
evaluation  activity. 
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1.3.1  Removal  of  Category  3  Part  Manufacturers  from  the  ACSEP  Evaluation 
Schedule 

The  Aircraft  Certification  Management  Team  (ACMT)  chartered  the  Effective  Resource 
Utilization  Team  (ERUT)  in  Fiscal  Year  (FY)  99  to  identify  work  functions  that  utilize  a 
high  number  of  work  hours.  The  ERUT  was  also  chartered  to  identify  strategies  to 
modify,  eliminate,  or  delegate  these  work  functions  in  order  to  shift  the  Service’s 
resources  to  higher  priority  work.  One  of  the  work  functions  identified  by  the  ERUT  was 
the  conduct  of  ACSEP  evaluations  at  non-priority  PAH’s.  The  ERUT  proposed  that 
ACSEP  evaluations  at  non-priority  PAH’s  be  discontinued.  In  its  place,  the  ERUT 
proposed  the  implementation  of  a  structured  process  for  PI  evaluation  of  these  PAH’s. 
Early  in  FY  2000,  the  FAA  decided  to  remove  Category  3  Part  manufacturers  fi'om  the 
ACSEP  evaluation  schedule.  A  Category  3  Part  is  defined  within  Order  8100.7A, 
Appendix  2,  page  5,  as  a  part  whose  “failure  would  have  no  effect  on  continued  safe 
flight  and  landing  of  the  aircraft.” 

Approximately  290  production  approval  holder  (PAH)  evaluations  were  removed  from 
the  ACSEP  evaluation  schedule  as  a  result  of  this  policy  change.  This  represents  a 
50  percent  reduction  in  the  total  number  of  ACSEP  evaluations  performed  each  year. 

The  Certificate  Management  of  Category  3  Part  manufacturers  is  now  accomplished  by 
Principal  Inspector  (PI)  audits. 

1.3.2  Initiated  The  Use  of  Revised  Forms  8100-4/8100-8 

To  further  the  data  analysis  and  presentation  efforts,  AIR-200  initiated  the  use  of  revised 
Forms  8100-4  and  8100-8.  The  evaluation  team  now  chooses  one  of  four  possible  survey 
responses,  or  a  combination  of  those  four  responses,  for  each  of  the  228  criteria.  The  four 
choices  are:  Procedures  in  place.  No  procedures.  Not  applicable,  or  Unable  to  evaluate. 
The  response  to  these  survey  questions  will  provide  greater  insight  into  the  significance 
of  recorded  issues  and  their  ranking  among  all  issues.  This  information  will  assist  the  PI 
in  focussing  attention  to  potential  problem  areas  that  may  not  have  been  previously 
apparent.  For  example,  suppose  that  out  of 250  issues  recorded  against  PC  holders,  only 
nine  were  recorded  against  system  element  3BE4  -  Software  Security.  A  first  impression 
would  be  that  this  is  not  very  significant.  Now  suppose  that  of  all  PC  holders,  only  10 
had  procedures  in  place  for  system  element  3BE4.  Of  those  ten  PC  holders,  nine  had 
issues  recorded  against  system  element  3BE4.  This  means  that  90  percent  of  PC  holders 
had  issues  against  system  element  3BE4.  This  should  alert  a  Pi’s  attention  to  a  PC  holder 
that  utilizes  procedures  for  system  element  3BE4. 

The  revised  survey  forms  were  not  implemented  until  the  FY  2000  evaluations  had 
already  begun.  The  information  obtained  from  them  will  not  be  incorporated  in  this 
report  since  they  were  not  used  for  the  complete  reporting  period.  They  were  used  for  all 
of  the  delegated  facility  evaluations  and  that  information  is  presented  in  this  report. 
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1.3.3  Completion  of  Statistical  Trending  Analysis 

FY1999  was  the  final  year  for  statistical  trending  analysis  of  ACSEP  data.  Previous 
ACSEP  data  showed  consistent  trends  with  little  variation.  Customer  feedback  (FAA  and 
PAH)  indicated  that  there  was  enough  trending  and  statistical  information  presented  in 
past  ACSEP  Annual  Reports  to  meet  their  requirements.  Further  presentation  of  trending 
data  would  not  be  useful  because  it  would  just  be  repetitious  of  previously  reported 
results.  Therefore,  data  will  now  be  presented  as  straight  counts  and  compared  to 
previously  reported  statistical  trends.  Presentation  of  straight  counts  will  provide  a  direct 
path  to  an  area  that  the  reader  may  have  interest  in. 

1.4  Overview  of  the  ACSEP  Activity 

The  transition  from  QASAR  to  ACSEP  occurred  in  FY  1993.  Figure  1-1  shows  the 
growth  of  the  program  from  FY  1994  to  FY  2000  (all  facilities  where  an  ACSEP 
evaluation  was  performed,  including  PPS  facilities,  are  shown  in  the  figure).  The 
evaluation  of  delegated  facilities  began  in  FY  1 998  after  the  release  of  Notice  N8 1 00. 1 3, 
Aircraft  Certification  Systems  Evaluation  Program  Criteria  for  Delegated  Facilities. 

From  FY  1993  through  FY  1998,  the  number  of  evaluations  performed  at  production 
approval  holders  increased  annually  at  an  average  of  24  percent.  The  growth  of  the 
program  was  facilitated  by  an  increase  in  the  number  of  qualified  manufacturing, 
engineering,  and  flight  test  personnel  fully  trained  to  perform  ACSEP  evaluations.  The 
reduction  in  the  number  of  ACSEP  evaluations  fi-om  FY  1999  to  FY  2000  is  the  result  of 
the  transition  of  Category  3  Part  manufacturers  from  ACSEP  to  PI  audits  and  the  full 
unplementation  of  Resource  Targeting.  Table  1-1  itemizes  the  population  of  various 
production  approval  holders'. 


'  Facilities  with  multiple  production  approvals  are  accounted  for  only  once  in  accordance  with  the 
following  order  of  precedence:  PC  (or  PCEX),  TSO,  APIS,  and  PMA. 
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Figure  7-7. — Growth  in  annual  ACSEP  evaluations. 


Table  7-7. — The  population^  ofPAHs  for  fiscal  years  1994  through  2000 


Fiscal  Year 

Parts 

Manufacturer 

Approval 

(PMA) 

Technical 
Standard  Order 
(TSO) 

Authorization 

Production 
^  Certificate 
(PC) 

Approved 
Production 
Inspection 
Systems  (APIS) 

Total  number  of 
Production 
Approval 
Holders  (PAH) 

1994 

1,140 

379 

74 

14 

1,607 

1995 

1,106 

309 

88 

5 

1,508 

1996 

1,413 

342 

70 

13 

1,838 

1997 

1,437 

364 

98 

8 

1,907 

1998 

1,211 

307 

98 

5 

1,621 

1999 

1,208 

306 

96 

5 

1,615 

2000 

1,229 

302 

109 

9 

1,649 

^  This  table  is  a  compilation  of  data  received  from  the  individual  directorates  and  is  included  in  this  report 
for  reference  only. 

^  Includes  PC  extensions. 
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Figure  1-2— Distribution  ofACSEP  evaluations  at  manufacturing  facilities  by  facility  type  — 

domestic  and  international  combined. 


The  distribution  ofACSEP  evaluations  among  the  various  facility  types  is  presented  in 
Figure  1-2.  Figure  1-2  shows  the  reduction  in  the  number  of  supplier  facilities  evaluated 
in  FY  1999  —  the  result  of  supplier  surveillance  being  conducted  through  PI  audits 
versus  AC  SEP.  As  presented  in  the  FY  1999  ACSEP  Annual  Report,  the  reduction  in  the 
number  of  evaluations  of  PC  holders,  PC  extensions,  APIS,  and  TSO  authorizations  is  a 
direct  result  of  Resource  Targeting  for  FY  1999.  The  number  of  evaluations  of  PMA 
holders  increased  to  a  number  that  was  consistent  with  both  the  population  of  PMA 
facilities  and  current  ACSEP  policy.  Any  future  increase  or  decrease  in  the  number  of 
PMA  holders  evaluated  will  reflect  solely  the  growth  or  decline  in  the  total  population  of 
PMA  holders.  The  reduction  in  the  number  of  FY2000  evaluations  is  a  direct  result  of 
the  transition  of  Category  3  Part  manufacturers  from  the  ACSEP  process. 

ACSEP  evaluations  were  conducted  by  the  Aircraft  Certification  Service's  four 
directorates.  Figure  1-3  shows  the  distribution  of  all  manufacturing  evaluations  among 
the  four  directorates. 
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Figure  1-3.— Distribution  ofACSEP  evaluations  at  maniffacturing  facilities  by  directorate  — 

domestic  and  international  combined. 


Table  1-2  lists  the  population  of  the  various  delegations.  The  distribution  of  the  ACSEP 
evaluations  among  the  varioiis  delegation  types  and  among  the  various  directorates  is 
shown  in  Figures  1-4  and  1-5  respectively. 


Table  1-2. — The  population'*  of  delegated facilities  for  fiscal  2000 


Fiscal  Year 

Designated 
Alteration  Station 
(DAS) 

Special  Federal  Aviation 
Regulation  No.  36  to 
CFR  part  121 
(SFAR-36) 

Delegation  Option 
Authorization 
(DOA) 

Total  number 
of  Delegated 
Facilities 

1998 

31 

24 

6 

61 

1999 

30 

22 

6 

58 

2000 

31 

13 

6 

50 

''  This  table  is  a  compilation  of  data  received  from  AIR- 100  and  is  included  in  this  report  for  reference 
only. 
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Figure  1-4.  Distribution  of  ACSEP  evoluations  at  delegated  facilities  by  delegation  type. 


Figure  1-5.— Distribution  of  ACSEP  evaluations  at  delegated  facilities  by  directorate. 


1.5  The  Data  Collected  During  an  ACSEP  Evaluation 

The  ACSEP  was  designed  to  determine  if  FAA  production  approval  holders  and 
delegated  facilities  are  complying  with  the  requirements  of  applicable  CFR  and  the 
procedures  established  by  these  facilities  to  meet  those  requirements.  It  also  surveys  the 
application  of  standardized  industry  practices  not  required  by  the  CFR  to  identify  national 
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issues  that  may  require  development  of  new  or  revised  regulations,  policy,  or  guidance. 
The  elements  of  the  evaluation  are  referred  to  as  criteria.  Data  is  collected  on 
noncompliance,  nonconformance,  and  applicability  with  respect  to  those  criteria. 

1.5.1  The  Various  Types  of  Issues 

During  an  ACSEP  evaluation,  the  actual  operating  practices  of  a  facility  are  compared  to 
the  CFR,  FAA-approved  data,  and  the  facility’s  internal  procedures.  Any  inconsistency 
discovered  (termed  issue  in  this  report)  is  classified  and  recorded.  An  issue  is  classified 
by  its  type  and  the  system  element  under  which  it  is  noted.  There  are  five  issue  t3q)es: 

Safety  Finding  -  an  issue  that  compromises  immediate  continued  operational 
safety. 

Systemic  Finding  -  an  issue  that  is  systemic  in  nature,  i.e.,  is  pervasive,  repeatable, 
or  represents  a  breakdown  in  the  quality  management  system.  For 
an  issue  to  be  categorized  a  finding,  it  must  also  be  a  noncompliance 
to  a  CFR  or  FAA-approved  data  (or  a  noncompliance  with  the 
procurement  instrument  when  a  facility  is  a  supplier). 

Systemic  Observation  -  an  issue  that  is  systemic  in  nature  and  is  a  noncompliance 
to  facility  procedures  that  are  not  FAA  approved. 

Isolated  Observation  -  an  issue  that  is  isolated  or  nonsystemic  in  nature,  i.e., 
isolated  to  a  particular  person  and/or  timeframe  and  does  not 
represent  a  breakdown  in  the  quality  management  system.  For  an 
issue  to  be  categorized  an  isolated  observation,  it  must  also  be  an 
isolated  noncompliance  to  a  CFR  or  FAA-approved  data  (or  a 
noncompliance  with  the  procurement  instrument  when  a  facility  is  a 
supplier). 

CFR-based  Observation  -  the  discovery  of  FAA-approved  data  that  is  inconsistent 
with  the  CFR. 

In  practice,  a  noncompliance/nonobservance  of  a  procedure  can  be  recorded  as  either  a 
finding  or  a  systemic  observation  based  solely  on  whether  the  procedure  was  FAA 
approved.  The  number  and  type  of  procedures  that  are  FAA-approved  varies  widely 
among  the  various  approval  types.  Additionally,  the  CFR  requirements  differ  among  the 
various  approval  types. 
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1.5.2  Issues  Classified  into  System  Elements 

The  second  form  of  classification  of  an  issue  is  the  system  element  under  which  it  is 
(Uscovered.  In  total,  there  are  17  system  elements  (listed  by  system  element  number  and 
title)  that  represent  a  quality  management  system  for  a  production  approval  holder: 

1  Organization  and  Responsibility  10  Supplier  Control 

2  Design  Data  Control  1 1  Nonconforming  Material 

3  Software  Quality  Assurance  12  Material  Handling/Storage 

4  Manufacturing  Processes  13  Airworthiness  Determination 

5  Special  Manufacturing  Processes  14  FAA  Reporting  Requirements 

6  Statistical  Quality  Control  (SQC)  15  Internal  Audit 

7  Tool  and  Gauge  16  Global  Production 

8  Testing  17  Manufacturing  Maintenance  Facility 

9  Nondestructive  Inspection 

There  are  10  system  elements  (listed  by  system  element  number  and  title)  that  represent  a 
quality  management  system  for  a  delegated  facility: 

1  Orgamzation  and  Responsibility  6  Project  Management 

2  Design  Data  Approval  7  Design  Change  Approval 

3  Testing  g  Conformity  Inspection 

4  Airworthiness  Certification  9  FAA  Notification 

5  Continued  Airworthiness  10  Audit 

1.5.3  System  Elements  Classified  into  Criteria 

Each  system  element  is  further  divided  into  “criteria.”  The  criteria  were  developed  with 
extensive  assistance  from  industry  in  order  to  fully  represent  the  detailed  areas  within 
each  of  the  system  elements.  A  process  also  exists  to  identify  potential  new  criteria 
should  the  existing  criteria  not  address  a  particular  functional  area  within  a  system 
element.  The  subclassification  of  issues  into  the  detailed  criteria  allows  the  FAA  to 
identify  specific  areas  of  concern  and  allows  industry  to  focus  corrective  action  on  these 
specific  areas  of  concern.  For  example,  the  supplier  control  system  element  is  composed 
of  16  individual  criteria.  Specific  areas  of  concern  that  may  be  identified  include:  the 
use  of  approved  suppliers;  periodic  evaluations  of  suppliers;  flowdown  of  applicable 
technical  and  quality  requirements  to  suppliers;  raw  material  verification;  and  others. 

Through  the  use  of  detailed  criteria  and  their  relevant  system  elements,  quality 
management  systems  can  be  evaluated  in  a  consistent  manner. 
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2.  Conclusions  based  on  the  Data 

Review  of  the  FY  2000  ACSEP  evaluation  data  supports  the  following  conclusions: 

•  There  was  one  safety  finding  recorded  at  a  delegated  facility  in  the  area  of  Design 
Data. 

•  The  majority  of  findings  and  observations  are  concentrated  within  a  few  system 
elements:  manufacturing  processes,  supplier  control,  tool  and  gauge,  design  data 
control,  nonconforming  material,  and  special  manufacturing  processes. 

•  An  area  of  special  focus  is  chosen  aimually  to  determine  if  increased  oversight  is 
required.  Software  Quality  Assurance  was  chosen  as  the  special  focus  for  this  fiscal 
year.  There  were  a  total  of  six  systemic  findings,  two  systemic  observations,  three 
isolated  observations,  and  one  CFR-based  observation.  These  were  equally 
distributed  amongst  the  specific  criteria  within  this  system  element.  No  definitive 
conclusions  can  be  drawn  on  these  results  at  this  time.  It  is  expected  that  the  number 
of  issues  recorded  in  this  area  will  increase  over  time  based  on  industry’s  greater 
reliance  on  software  driven  technologies.  As  more  manufacturing  systems  develop  a 
greater  dependence  on  software  driven  systems,  this  area  will  require  greater 
diligence  in  its  surveillance  by  quality  assurance  and  audit  persoimel.  These 
personnel  will  have  to  ensure  that  they  remain  well  versed  in  the  current  software 
technologies  and  software  system  audit  techniques. 


Aircraft  Certification  Systems  Evaluation  Program  FY  2000  Report 


16 


3.  Data  Analysis  —  Manufacturing  Facilities 

3.1  Safety  Related  Findings 

Of  the  61 1  findings  and  observations  recorded  at  production  approval  holder  facilities  in 
FY  2000,  none  identified  immediate  safety  concerns.  During  an  ACSEP  at  a  Designated 
Alteration  Station  (DAS),  one  Safety  Finding  was  recorded  in  the  area  of  Design  Data 
Approval  (specifically  criteria  3D  1 -Control  of  Type  Design  Data)  for  failure  to  include 
required  safety  system  information  in  the  STC  data  package. 

3.2  Systemic  Findings 

There  were  294  systemic  findings  reported  in  FY  2000.  At  least  one  systemic  finding 
was  recorded  at  33  percent  of  the  production  approval  holders  evaluated  in  FY  2000.  Of 
all  of  the  systemic  issues  recorded,  79  percent  were  recorded  within  only  six  of  the 
system  elements.  These  six  system  elements  are  displayed  in  Figure  3-1. 


,,  Jh..:  '%■  i*.-, 

Manufacturing  Processes 


Supplier  Control 


Nonconforrrtng  Material^^S 
Design  Data  ControlMMMMiMiMii 
Tool  and  GuageB 

Spedal  Manufa<aurtng-;:Pfi^i^^sl^^^Mai 


0%  5%  10%  15%  20%  25%  30% 


Per<»ntage  of  systemic  issues 


Figure  3-1. —  Systemic  findings  -  all  facility  types. 
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3.3  Systemic  Observations 

There  were  147  systemic  observations  reported  in  FY  2000.  At  least  one  systemic 
observation  was  recorded  at  9  percent  of  the  production  approval  holders  evaluated  in 
FY  2000.  Of  all  of  the  systemic  observations  recorded,  83  percent  were  recorded  within 
only  six  of  the  system  elements.  These  six  system  elements  are  displayed  in  Figure  3-2. 


Supplier  Control 


Manufecturing  Processes 


Design  Data  Control 
Noncon^rrring  Material 
Organization  and  Re^mnsibitity 


Percentage  of  systemic  issues 


Figure  3-2. —  Systemic  observations  —  all  facility  types. 
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3.4  Isolated  Observations 

There  were  128  isolated  observations  reported  in  FY  2000.  At  least  one  isolated 
observation  was  recorded  at  7  percent  of  the  production  approval  holders  evaluated  in 
FY  2000.  Of  all  of  the  isolated  observations  recorded,  84  percent  were  recorded  within 
only  six  of  the  system  elements.  These  six  system  elements  are  displayed  m.  Figure  3-3. 


Manutecturing  Processes 
Supplier  Control 
Nonconforming  Material 
Special  Manufacturing  Processes 
Tool  and  Quage 
Material  Handling/Stonage 


Percentage  of  Isolated  issues 


Figure  3-3. —  Isolated  observations  -  all  facility  types. 

3.5  CFR-Based  Observations 

There  were  43  CFR-based  observations  reported  in  FY  2000.  Table  3-1  lists  those 
system  elements  where  the  CFR-based  observations  were  reported.  There  were  19  CFR- 
based  observations,  with  Manufacturing  Processes  having  the  greatest  number  of  issues, 
reported  in  FY  1999. 
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Table  3-1. — CFR-based  observations 


Domestic 

Number  of  CFR-based 
observations  reported 

Design  Data  Control 

14 

Manufacturing  Processes 

6 

Supplier  Control 

6 

Organization  &  Responsibility 

5 

Testing 

3 

Statistical  Quality  Control  (SQC) 

3 

Software  Quality  Assurance 

2 

Special  Manufacturing  Processes 

1 

Nonconforming  Material 

1 

Airworthiness  Determination 

1 

FAA  Reporting  Requirements 

1 

3.6  System  Element  Issues 

3.6.1  Similarity  Among  Approval  Types 

Tables  3-2  through  3-4  show  tihe  most  prevalent  issues,  as  defined  by  the  total  number  of 
systemic  findings  and  observations  combined,  for  each  of  the  approval  types.  There  were 
no  issues  recorded  for  the  two  APIS  ACSEPs  performed  this  year.  Table  3-5  shows  the 
most  prevalent  issues  for  all  of  the  approval  types  combined.  It  is  apparent  from  this 
presentation  that  the  distribution  of  issues  for  all  of  the  approval  types  combined  is 
similar  to  that  for  any  individual  approval  type  alone.  Table  3-6  summarizes  the  data 
contained  in  the  figures  by  comparing  the  most  prevalent  issues  among  the  various 
facility  types. 

Please  note  that  direct  comparison  of  the  approval  types  cannot  be  done  with  these  charts. 
As  revealed  in  the  FY1999  Annual  ACSEP  Report,  the  proportion  of  facilities  with 
systemic  issues  is  strongly  related  to  system  complexity.  Because  there  are  significant 
differences  in  system  complexity  among  the  various  approval  types,  these  charts  cannot 
be  used  to  compare  compliance  between  approval  types. 
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Table  3-2.— Counts  ofPMA  issues. 


System  Element 


Organization  and 
Responsibility 


Design  Data 
Control 


Software  Quality 
Assurance 


Manufacturing 

Processes 


Special 

Manufacturing 

Processes 


Statistical  Quality 
Control 


Tool  &  Gauge 


Testing 


Nondestructive 

Inspection 


Supplier  Control 


Nonconforming 

Material 


Material 

Handling/Storage 


Airworthiness 

Determination 


FAA  Reporting 
Requirements 


Internal  Audit 


Global  Production 


Manufacturer’s 

Maintenance 

Facility 


TOTAL 


Isolated 

Observations 


6 


CFR-Based 

Observations 


2 
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Table  3-3. — Counts  of  PC  issues. 


System  Element 


Organization  and 
Responsibility 


Design  Data 
Control 


Software  Quality 
Assurance 


Manufacturing 

Processes 


Special 

Manufacturing 

Processes 


Statistical  Quality 
Control 


Tool  &  Gauge 


Testing 


Nondestructive 

Inspection 


Supplier  Control 


Nonconforming 

Material 


Material 

Handling/Storage 


Airworthiness 

Determination 


FAA  Reporting 
Requirements 


Internal  Audit 


Global  Production 


Manufacturer’s 

Maintenance 

Facility 


TOTAL 


Systemic 
Fmlings  ,, 


Isolate 

Observations 


0 


CFR-Based 

Obsfervations 


1 
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System  Element 


Organization  and 
Responsibility 


Design  Data 
Control 


Software  Quality 
Assurance 


Manufacturing 

Processes 

Special 

Manufacturing 

Processes 


Statistical  Quality 
Control 


Tool  &  Gauge 


Testing 


Nondestructive 

Inspection 


Supplier  Control 


Nonconforming 

Material 


Material 

Handling/Storage 


Airworthiness 

Determination 


FAA  Reporting 
Requirements 


Internal  Audit 


Global  Production 


Manufacturer’s 

Maintenance 

Facility 


TOTAL 


Table  3-4.— Counts  ofTSOA  issues. 


Systemic  Systemic  .  .  hoisted 

Findings  Observations  Observations 


2  0 


GFR-Based 

Observations 


2 


0 

0 

1 

0 

17 

16 
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Table  3-5. — Counts  of  all  issues. 


System  Element 

Systemic 

F'ihdings 

,;.r:  Systemic,. 
Observations 

Isolated 
‘  Observations 

CFR-Based 

Observations 

Organization  and 
Responsibility 

8 

10 

7 

5 

Design  Data 
Control 

30 

13 

7 

14 

Software  Quality 
Assurance 

6 

2 

3 

1 

Manufacturing 

Processes 

78 

31 

35 

6 

Special 

Manufacturing 

Processes 

24 

4 

11 

1 

Statistical  Quality 
Control 

3 

0 

2 

3 

Tool  &  Gauge 

29 

22 

19 

0 

Testing 

7 

2 

4 

3 

Nondestructive 

Inspection 

8 

2 

3 

0 

Supplier  Control 

42 

34 

10 

6 

Nonconforming 

Material 

30 

12 

11 

1 

Material 

Handling/Storage 

9 

6 

10 

0 

Airworthiness 

Determination 

1 

1 

0 

1 

FAA  Reporting 
Requirements 

4 

1 

2 

1 

Internal  Audit 

11 

7 

2 

0 

Global  Production 

0 

1 

0 

Manufacturer’s 

Maintenance 

Facility 

3 

0 

1 

0 

TOTAL 

294 

147 

128 

42 
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Table  3-6.— Summary  of  the  most  prevalent  systemic  issues  -  FY 2000 


System  Element 

ALL 

PC 

PMA 

TSOA 

Manufacturing  Processes 

X 

X 

X 

X 

Supplier  Control 

X 

X 

X 

X 

Design  Data  Control 

X 

X 

X 

X 

Tool  &  Gauge 

X 

X 

X 

X* 

Nonconforming  Material 

X 

X 

X 

X 

Special  Manufacturing  Processes 

X 

X 

X 

Internal  Audit 

X* 

X  =  One  of  the  top  six  systemic  issues 


J 


*  =  Tied 


Leading  issues 
for  industry 


Aircraft  Certification  Systems  Evaluation  Program  FY  2000  Report 


25 


A  five-year  comparison  of  the  most  frequently  cited  system  elements  with  systemic 
issues  {see  Table  3-7)  indicates  that  there  have  been  only  minor  variations  in  the  order  of 
occurrence  at  the  system  element  level.  The  various  approval  holders  appear  to  have 
similar  key  issues.  With  the  exception  of  some  minor  shifting  in  position,  the  top  issues 
have  remained  the  top  issues  over  the  five  years. 


Table  3-7. — Most  frequently  cited  system  elements  with  systemic 
issues  -  FY  1996  through  FY 2000 


ALL  APPROVAL  TYPES 

Manufacturing  Process _ 

Supplier  Control _ 

Tool  and  Gauge _ 

Design  Data  Control _ 

Nonconforming  Material _ 

Material  Handling/Storage _ 

PC 

Manufacturing  Process _ 

Tool  and  Gauge _ 

Special  Manufacturing  Processes 

Design  Data  Control _ 

Nonconforming  Material _ 

Supplier  Control _ 

Material  Handling/Storage _ 

PMA 

Manufacturing  Process _ 

Supplier  Control _ 

Tool  and  Gauge _ _ 

Design  Data  Control _ 

Nonconforming  Material _ 

TSO 

Manufacturing  Process _ 

Supplier  Control _ 

Nonconforming  Material _ 

Design  Data  Control _ 

Tool  and  Gauge _ 

Material  Handling  and  Storage 


Annual  System  Element  Rank 
FY  FY  FY  FY  FY 

1996  1997  1998  1999  2000 


1  1.1  1  1 
2  2  2  2  2 

3  3  3  4  3 

4  4  3  3  4 

5  6  5  5  5 

6  4  5  6  7 


2  1111 
1  3  8  5  2 

4  4  3  3  3 

5  6  3  6  4 

8  6  3  3  5 

3  2  3  2  6 

8  4  2  8  12 


2  1111 
1  2  2  2  2 

4  3  3  4  3 

4  5  3  3  4 

3  4  5  5  5 


1112  1 
2  2  2  1  2 

6  5  4  5  3 

3  4  4  4  4 

4  3  4  3  5 

5  8  3  5  7 
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3.7  Analysis  of  Evaluation  Criteria 

The  following  subsections  contain  lists  of  the  most  significant  criteria  issues  at  any  given 
facility  type.  This  data  can  be  used  by  industry  to  focus  corrective  action  and  by  the  FAA 
for  resource  allocation  initiatives.  The  data  is  presented  in  three  forms:  a  view  of 
industry  as  a  whole  listed  by  type  of  issue  —  systemic  or  isolated;  a  focus  on  individual 
approval  types  in  which  systemic  issues  are  separated  by  approval  type;  and  a  summary 
of  comparisons  among  the  approval  types.  For  clarity,  only  the  top  issues  are  reported  in 
these  subsections. 

3.7.1  A  View  of  Industry 

This  subsection  lists  the  most  prevalent  criteria  issues  within  the  industry  as  a  whole. 

The  data  from  all  of  the  ACSEP  evaluations  performed  in  FY  2000  are  first  presented 
pooled  together  (Table  3-8).  The  table  column  titled  “Percent  of  All  Facilities”  presents 
the  proportion  of  facilities  evaluated  that  had  issues  recorded. 

3.7.1. 1  Systemic  findings  and  observations 

The  1 1  evaluation  criteria  most  frequently  recorded  with  systemic  issues  are  presented  in 
Table  3-8.  These  1 1  criteria  accounted  for  almost  39  percent  of  all  reported  systemic 
issues.  As  a  group,  they  occurred  at  52  percent  of  the  facilities  with  recorded  issues. 


Table  3-8. — Ten  most  reported  criteria  with  systemic  issues 


Rank 

Criteria 

Description 

Number  6f 
‘Systentic  ? 
issues 

Percent  bf 
Systemic 
Issues 

Percent 
of  All 
Facilities 

1 

10Q1 

Initial  &  periodic  evaluations  of  suppliers 

23 

5% 

15% 

2 

5Q3 

Performing  special  processes  in 
accordance  with  process  specifications 

20 

5% 

13% 

3 

4Q1 

Inspection  methods  and  plans 

18 

4% 

12% 

11Q1 

Control  of  nonconforming  products 

16 

4% 

10% 

4P9 

Completed  product/part  identification 

15 

3% 

10% 

6 

10Q10 

Receiving  inspection 

15 

3% 

10% 

B 

15M1 

Internal  auditing  program 

14 

3% 

10% 

o 

4Q5 

Inspection  records 

14 

3% 

10% 

H 

2E7 

Design/Technical  data  document 
control 

3% 

10% 

10 

7Q1 

Approval/inspection  of  tools  and  gauges 

12 

3% 

8% 

11 

11Q2 

Permanent  identification  of  scrap 
material 

11 

2% 

8% 
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Table  3-9  illustrates  that  many  of  the  most  significant  systemic  issues  have  been 
significant  for  the  last  five  years.  The  table  lists  the  top  ten  most  cited  criteria  for  the  last 
five  years.  The  columns:  FY  2000,  FY  1999,  FY  1998,  FY  1997,  and  FY  1996  indicate 
whether  the  criteria  was  a  top  issue  for  that  year.  Six  of  the  ten  have  been  the  top  issues 
for  four  or  more  of  the  last  five  years.  Note  that  the  criteria  are  not  presented  according 
to  ranking.  They  are  in  random  order. 


Table  3-9. — Five-year  comparison  of  most  predominant  systemic  issues  —  by  criteria. 


Criteria 

FY 

2000 

FY 

1999 

FY 

1998 

FY 

1997 

FY 

1996 

10Q1 

Initial  &  periodic  evaluations  of  suppliers 

□ 

a 

X 

X 

X 

4P9 

Completed  product/part  identification 

D 

□ 

X 

X 

X 

15M1 

Internal  auditing  program 

D 

□ 

X 

X 

X 

11Q1 

Control  of  nonconforming  products 

D 

□ 

X 

X 

X 

5Q3 

Accord  with  process  specifications 

■ 

□ 

X 

X 

X 

4P4 

Work  instructions  control  manufacturing 
processes 

■ 

□ 

X 

X 

10Q10 

Receiving  inspection 

□ 

D 

X 

X 

10Q5 

Flow  down  of  technical  &  quality  requirements 

■ 

X 

X 

10Q8 

Verification  of  raw  material 

■ 

X 

X 

4Q5 

Inspection  records 

D 

D 

X 

X 

^  _  Criteria  within  the  top  tenth  percentile  for  the  fiscal 
year 

"blank"  =  Criteria  within  the  lower  90th  percentile  for  the  fiscal 
year 
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3.7.2  A  Facility  Focus 

This  section  lists  the  criteria  issues  separated  by  approval  type  {Tables  3-10  to  3-12). 

This  allows  the  reader  to  focus  on  the  issues  pertinent  to  a  particular  approval  type 
without  bias  from  the  other  approval  types.  For  example,  the  data  from  the  relatively  few 
PC  holders  is  not  skewed  by  the  data  from  the  much  larger  population  of  PMA  holders. 

For  clarity,  only  the  top  issues  are  reported  in  this  section. 


Table  3-10. — Predominant  systemic  issues  —  PC  holders 


Rank 

Criteria 

Description 

Number  of 
Systemic 
issues 

Percent  of 
Systemic  V 
Issues  for 
PC  Hbideis 

Percentof 
PCJHolders 
wlUi  issues 

1 

— 

5Q3 

Accord  with  process 
specifications 

9 

5% 

29% 

2 

2E7 

Design/Technical  data 
document  control 

5 

3% 

17% 

3 

7Q1 

Approval/inspection  of  tools 
and  gauges 

5 

3% 

17% 

H 

Control  of  nonconforming 
products 

4 

2% 

13% 

5 

10Q1 

Initial  &  periodic  evaluations  of 
suppliers 

3 

2% 

10% 

5 

4Q1 

Inspection  methods  and  plans 

3 

2% 

10% 

5 

10Q10 

Receiving  inspection 

3 

2% 

10% 

5 

15M1 

Internal  audit 

3 

2% 

10% 

5 

4Q3 

issuance  of  inspection  stamps 

3 

2% 

10% 

5 

7Q12 

Calibration  Records 

3 

2% 

10% 
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Table  3-1 1. — Predomincmt  systemic  issues  —  PMA  holders 


Rank 

Criteria 

Description '  ■  ■  ■ 

Number  of 
Systemic  ^ 
issues 

Percent  of  Total 
Systemic 
issues  for  ^ 
PMA  Holders 

Percent 
of  PMA 
Hoideis 

■ 

4P9 

Completed  product/part 
identification 

15 

5% 

16% 

1 

10Q1 

Initial  &  periodic  evaluations  of 
suppliers 

15 

5% 

16% 

3 

4Q1 

Inspection  methods  and  plans 

12 

4% 

13% 

■ 

11Q1 

Control  of  nonconforming 
products 

9 

3% 

10% 

H 

5Q3 

Accord  with  process 
specifications 

9 

3% 

10% 

H 

2E7 

Design/Technical  data 
document  control 

8 

3% 

9% 

7 

Verification  of  raw  material 

8 

3% 

9% 

Table  3-12. — Predominant  systemic  issues  —  TSO  authorization  holders 


Rank  Criteria  Description 


1  1 1 0Q1 0 1  Receiving  inspection 


Initial  &  periodic  evaluations 
of  suppliers 


Work  instructions  control 
manufacturing  processes 


Internal  auditing  program 


Inspection  methods  and 
plans 


Use  of  approved  suppliers 


Inspection  methods  and 
plans 


Control  of  nonconforming 
products 


Humber  of 
Systemic 
Issues 


6 


5 


Percent  6f 
Total  Systemic 
Issues  fortSp 
Authoriz^bns 

Percent^  ISO 
Autboilzatior^ 

5% 

17% 

4% 

14% 

3% 

11% 

3% 

11% 

2% 

9% 

2% 

9% 

2% 

9% 

2% 

9% 
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3.8  Software  Quality  Assurance 

Software  Quality  Assurance  was  chosen  as  the  selected  analysis  for  this  fiscal  year. 

There  were  a  total  of  12  issue  recorded  within  the  Software  Quality  Assurance  element 
for  this  fiscal  year.  This  accounted  for  2  percent  of  the  total  issues  recorded.  The  issues 
were  evenly  distributed  amongst  the  various  approval  types.  There  were  a  total  of  six 
systemic  findings,  two  systemic  observations,  three  isolated  observations,  and  one  CFR- 
based  observation.  These  were  equally  distributed  amongst  the  specific  criteria  within 
this  system  element.  No  definitive  conclusions  can  be  drawn  on  these  results  at  this  time. 
It  is  expected  that  the  number  of  issues  recorded  in  this  area  will  increase  over  time  based 
on  industry’s  greater  reliance  on  software  driven  technologies.  As  more  manufacturing 
systems  develop  a  greater  dependence  on  software  driven  systems,  this  area  will  require 
greater  diligence  in  its  surveillance  by  quality  assurance  and  audit  personnel.  These 
personnel  will  have  to  ensure  that  they  remain  well  versed  in  the  current  software 
technologies  and  software  system  audit  techniques.  This  subject  will  be  addressed  again 
in  the  ACSEP  FY  2001  Annual  Report. 

3.9  Delegated  Facilities 

This  was  the  third  year  that  data  was  collected  for  facilities  with  engineering  delegation 
authority.  Delegated  facilities  include  Designated  Alteration  Stations  (DAS),  Special 
Federal  Aviation  Regulation  No.36  (SFAR-36)  facilities,  and  Delegation  Option 
Authorization  (DOA)  facilities.  For  this  fiscal  year,  17  systemic  findings  (including  one 
safety  finding),  5  isolated  observations,  and  3  FAR-based  observations  were  recorded.  A 
summary  of  the  data  follows. 

3.9.1  Designated  Alteration  Stations  (DAS)  Facilities 

Table  3-13  presents  the  breakdown  of  systemic  issues  recorded  for  DAS  facilities. 
ACSEP  evaluations  were  performed  at  8  DAS  facilities.  There  were  a  total  of  15 
systemic  issues  recorded.  The  two  most  prevalent  issues  were  in  the  areas  of  Conformity 
Inspection  and  Design  Data  Approval.  One  Safety  Finding  was  recorded  in  the  area  of 
Design  Data  Approval  (specifically  criteria  3D  1 -Control  of  Type  Design  Data)  for  failure 
to  include  required  safety  system  information  in  the  STC  data  package.  Table  3-14 
presents  the  breakdown  of  isolated  issues  recorded  for  DAS  facilities.  Three  FAR-based 
issues  were  recorded.  Two  were  in  the  area  of  Airworthiness  Certification  and  one  was  in 
the  area  of  FAA  Notification. 
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Table  3-13. — DAS  systemic  issues. 


Cumulative 
percent  of 


Percent 
of  all 
DAS 
facilities 
that  had 
systemic 
issues 


Percent  of 
DAS 
facilities 
where  the  i 
criteria 

applicable 

Percent  of 
DAS 
facilities 
where  the 
criteria 
was 

applicable 
and  bad 
systemic 
issues 

50% 

50% 

75% 

33% 

100% 

25% 

50% 

25% 

75% 

17% 

75% 

17% 

88% 

14% 

88% 

14% 

Table  3-14. — DAS  isolated  issues. 


1 

3D1 

2 

7D1 

3 

1D2 

3 

1D12 

3 

8D2 

Percent  of 
all  isolated 
DAS  issues 

Cumulative 
percent  of 
all  isolated 
DAS  issues 

Percent 
of  all 
:?:  DAS  ■ 

facilities 
that  had 
isolated 
:  issues 

Percent  of 
DAS 
facilities 
where  the 
criteria 
was 

applicable 

Percent  of 

facilities 
■  ■■  where|the  ^  ■ 
criteria 

applicable 
and  had 
isolated 
issues 

20% 

20% 

13% 

100% 

13% 

20% 

40% 

13% 

63% 

20% 

20% 

60% 

13% 

100% 

13% 

20% 

80% 

13% 

100% 

13% 

20% 

100% 

13% 

100% 

13% 
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3.9.2  Special  Federal  Aviation  Regulation  No.  36  (SFAR-36)  Facilities 

ACSEP  evaluations  were  performed  at  3  SFAR-36  facilities.  There  were  no  issues 
recorded. 

3.9.3  Delegation  Option  Authorization  (DOA)  Facilities 

There  was  one  ACSEP  conducted  at  a  DOA  facility.  Two  systemic  issues  and  no  isolated 
issues  were  recorded.  One  Systemic  Finding  was  recorded  in  the  area  of  Design  Data 
Approval  (specifically  criteria  3D  1 -Control  of  Type  Design  Data)  and  one  Systemic 
Finding  was  recorded  in  the  area  of  Design  Change  Approval  (specifically  criteria  4D1- 
Control  of  Changes  To  Type  Design  Data). 

3.9.4  A  Comparison  For  the  Last  Three  Years 

There  were  37  systemic  findings  recorded  in  FY  1998,  19  systemic  findings  recorded  in 
FY  1999,  and  17  systemic  findings  recorded  in  FY  2000  at  delegated  facilities.  The 
distribution  of  findings  amongst  the  three  tj^es  of  facilities  for  each  year  was  similar. 

The  distribution  of  systemic  findings  within  the  specific  system  elements  was  also 
similar.  A  presentation  of  the  actual  ranking  of  the  systemic  issues  within  the  system 
elements  for  the  last  three  years  would  be  ambiguous  because  of  the  small  number  of 
findings  recorded  and  because  specific  8100-8  survey  data  was  not  recorded  until  this 
year. 
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4.  Improvement  Emphasis 

The  goal  of  the  ACSEP  is  to  support  continuing  operational  safety  and  promote 
continuous  improvement. 

4.1  Industry  Feedback 

As  part  of  the  ACSEP  Quality  Improvement  Program,  a  performance  feedback  report 
(FAA  Form  8100-7,  FAA  ACSEP  Evaluation  Feedback  Report)  is  provided  to  each 
individual  organization  when  notified  that  an  evaluation  is  scheduled  to  take  place.  Each 
facility  evaluated  is  requested  to  use  this  report  to  critique  the  FAA  ACSEP  evaluation 
process.  The  feedback  report  is  used  to  record  the  facility’s  impression  for  each  step  of 
the  evaluation,  firom  notification  to  the  post-evaluation  conference.  A  question 
concerning  the  professionalism  of  the  ACSEP  evaluation  team  is  also  included  on  the 
report.  The  facility’s  management  is  encouraged  to  complete  the  report  and  return  it  for 
analysis.  Feedback  reports  were  returned  by  44  percent  of  the  facilities. 

Overall,  the  feedback  was  very  good.  As  with  the  previous  year,  greater  than  99  percent 
of  the  responses  were  “Satisfactory”  or  better  (see  Figure  4-1).  Figure  4-2  gives  the 
average  scores  for  each  of  the  feedback  categories  measured  and  an  overall  average.  The 
data  presented  remains  consistent  fi'om  the  previous  years. 

The  feedback  report  also  allows  for  the  inclusion  of  comments/suggestions.  The 
comments/suggestions  dealt  primarily  with  the  issues  of  scheduling,  providing  materials 
to  the  facility  prior  to  the  ACSEP  team’s  arrival,  more  PI  involvement.  Examples  of 
comments/suggestions  submitted  include: 

•  Would  like  more  advanced  notice  of  the  audit. 

•  Would  like  a  PI  visit  before  the  audit. 

•  Would  like  more  PI  visits. 

•  Provide  the  in-brief  slides  prior  to  the  team’ s  arrival. 

•  Provide  a  preliminary  copy  of  FAA  Form  8 1 00-4,  ACSEP  Survey  Sheet  for 
Production  Approval  Holders  to  the  facility  evaluated. 

•  Auditors  required  better  training. 
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Figure  4-1. — Distribution  of  industry  feedback. 


Timeliness 
Coordination/Planning 

Communication 
Presentation 
Purpose  of  evaluation  explained 

Explanation  of  findings/observations 
Resolution  of  issues 
Communication 
Explanation  of  executive  summary 
Explanation  of  follow-up  actions 
Team  professionalism 
Overall  technical  skills  of  the  ACSEP  team 
Overall 

Unsatisfactory  Poor  Satisfactory  Good  Excellent 


Figure  4-2. — ACSEP  as  graded  by  industry. 
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4.2  Lessons  Learned 

An  additional  part  of  the  continuous  improvement  process  is  the  gathering  and  analyzing 
of  lessons  learned  that  the  evaluation  team  documented  at  the  conclusion  of  each  ACSEP 
evaluation.  Each  ACSEP  evaluation  team  submits  a  “lessons  learned”  form  that  records 
the  team’s  general  assessment  of  the  evaluation,  difficulties  with  the  order,  system 
elements  not  evaluated,  and  any  proposed  new  criteria.  Figure  4-3  through  figure  4-6 
show  the  trend  in  these  lessons  learned  from  FY  1995  to  FY  2000. 


Figure  4-3. — Trend  of  lessons  learned— favorable  experiences. 


100% 

80% 

60% 

I 

j  40% 

! 

j 

I  20% 

I 

i  0% 


FY’95  FY'96  FY'97  FY’98 


FY’99 


FY'OO 


Figure  4-4. — Trend  of  lessons  learned — no  difficulties  with  Order  8100. 7 
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Figure  4-5. — Trend  of  lessons  learned — evaluation  completed. 


Figure  4-6. — Trend  of  lessons  learned — no  new  criteria  needed. 


The  percentage  of  teams  reporting  favorable  experiences  was  consistent  from  last  year. 
There  was  an  increase  in  the  percentage  of  teams  having  difficulties  using  the  order.  This 
is  directly  attributable  to  the  introduction  of  the  new  8 1 00-4  survey  form  and  placing  the 
Form  8100-6  in  Order  8120.2B.  The  percentage  of  teams  having  difficulties  using  the 
order  is  expected  to  decrease  as  teams  develop  greater  familiarity  with  the  new 
Form  8100-4.  There  was  a  slight  decrease  in  the  percentage  of  evaluations  completed. 
This  was  primarily  due  to  time  constraints  related  to  scheduling  and  weather  issues.  As 
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in  previous  years,  the  evaluation  teams  did  not,  as  a  whole,  require  the  need  for  new 
criteria. 

Figure  4-7  presents  the  number  of  ACSEPs  with  system  elements  not  completed.  The 
total  number  of  system  elements  not  evaluated  significantly  decreased  from  the  previous 
year.  This  is  consistent  with  the  total  reduction  in  the  number  of  ACSEPs  performed  this 
year.  Internal  Audit  dropped  from  second  to  ninth  in  ranking  for  system  elements  not 
evaluated.  This  implies  that  teams  have  recognized  the  importance  of  evaluating  this 
element  and  are  ensuring  that  it  is  adequately  addressed  during  the  evaluation. 


Statistical  Quality  Control  (SQC) 

Manufacturing  Maintainence  Facility 
Airworthiness  Determination 
Global  Production 
Nonconforming  Material 
FAA  Reporting  Requirements 
Nondestructive  Inspection  (NDI) 

Organization  &  ResponsibilKy 
Internal  Audit 
Software  Quality  Assurance 
Material  Handling/Storage 
Special  Manufacturing  Process 
Testing 
Design  Data  Control 
Supplier  Control 
Manufacturing  Process 
Tool  And  Gauge 

01  2345678 


Number  of  ACSEP  evaluations  with  subsystems  not  completed 


Figure  4-7. —  Distribution  of  subsystems  not  evaluated. 


Table  4-1  presents  a  detailed  breakdown  of  comments  received  with  the  Lessons  Learned. 
There  was  an  increase  in  the  response  to  “Time  scheduled  at  facility  was  too  short  or  too 
long.”  Two  significant  comments  were  received  for  this  reporting  period.  The  first  is 
that  there  were  a  marked  number  of  complaints  that  the  instructions  for  completing  the 
Form  8100-6  are  not  included  in  Order  8100.7A.  The  instructions  are  currently  in 
Order  8120.2  in  response  to  a  Certificate  Management  Team  (CMT)  recommendation. 
This  requires  the  team  to  carry  an  additional  piece  of  documentation  with  them  to  the 
audits.  The  second  significant  comment  was  from  teams  noting  that  they  felt  the  survey 
Form  8100-4  was  not  clear  and  unnecessary.  These  comments  were  also  reflected  in  their 
response  to  “difficulties  using  the  order.”  As  stated  previously,  teams  will  become  more 
comfortable  with  this  form  as  they  use  it  The  comment  that  they  feel  it  is  not  necessary 
can  be  attributed  to  the  teams  lack  of  imderstanding  for  the  purpose  of  the  information 
requested  on  the  form.  The  information  is  not  being  used  to  assist  in  the  specific  ACSEP 
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of  the  facility,  rather  it  is  being  collected  to  assist  in  the  national  survey  of  all  PAHs 
subject  to  an  ACSEP  for  the  given  reporting  period. 


Table  4-1. — Comments  received  from  lessons  learned  sheets 


General  Issues/Comments 

FY96 

FY'97 

FY'98 

FY'99 

FY'OO 

Time  scheduled  at  facility  was  too  short  or  to 
long 

6% 

5% 

5% 

3% 

7% 

Computer  or  ACSEP  software  issues 

0% 

0% 

3% 

1% 

2% 

Logistics;  no  escorts  or  QC  mgr.,  facility  not 
notified 

0% 

2% 

1% 

0% 

1% 

QC  Manual:  incomplete,  outdated,  conflicts 
with  other  procedures 

1% 

1% 

0% 

0% 

1% 

Production  is  very  low,  inactive,  or 
inappropriate  for  audit 

2% 

1% 

0% 

1% 

2% 

Management  defensive/uncooperative 

n/a 

1% 

0% 

mm 

0% 

ISO  9000  certification  better  prepared  the 
facilities  for  ACSEP  evaluation 

1% 

1% 

0% 

■n 

1% 

Recommend  extending  evaluation  frequency 

1% 

1% 

0% 

1% 

0% 

Misc.  other  issues 

2% 

2% 

3% 

1% 

1% 

Difficulty  with 

Fy’96 

Fy'97 

FY'98 

FY'99 

FY’OO 

Criteria;  add,  incorrect,  or  system  element 
issues 

5% 

4% 

2% 

2% 

2% 

ACSEP  too  big  for  facility 

2% 

0% 

1% 

1% 

1% 

Observations  &  findings;  confusion  with 
definitions 

1% 

0% 

0% 

1% 

1% 

Confusion  about  recording  multiple 
occurrences  of  findings  or  observations 

1% 

1% 

0% 

1% 

0% 

Instructions  for  Form  8100-6  not  in 

Order  8100.7A 

n/a 

n/a 

n/a 

n/a 

4% 

Form  8100-4  not  clear/not  necessary 

n/a 

n/a 

n/a 

a 
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APPENDIX  A.  HISTORY  AND  BACKGROUND  OF  ACSEP 

A1.  Background 

The  ACSEP  was  developed  as  a  result  of  numerous  years  of  experience  with  Quality 
Assurance  Systems  Analysis  Review  (QASAR)  audits  and  observations  made  during  an 
interim  audit  program  called  “Operation  SNAPSHOT.”  Maintaining  consistency  with 
new  FAA  policies  and  regulations,  with  regard  to  the  certificate  management  process, 
was  also  a  consideration  for  the  establishment  of  ACSEP.  The  intent  was  to  establish  a 
surveillance  system  that  would  meet  the  needs  and  requirements  of  the  FAA  and  industry, 
while  incorporating  standardized  evaluation  practices  and  techniques  consistent  with  the 
aircraft  manufacturing  environment  and  internationally  recognized  guidelines.  The 
evaluation  criteria  were,  in  part,  developed  in  conjunction  with  the  Aerospace  Industries 
Association  and  General  Aviation  Manufacturer's  Association.  By  design,  ACSEP  will 
support  continued  operational  safety  in  an  ever  changing  aircraft  manufacturing 
environment  (e.g.,  new  technologies,  automation,  and  co-production)  through  recurring 
evaluations  of  facilities’  quality  management  systems  and  tracking  and  trending  areas  for 
improvement. 

A2.  Overview 

ACSEP  is  an  Aircraft  Certification  Service  program.  The  Production  and  Arworthiness 
Certification  Division,  AIR-200,  is  the  national  focal  point  for  the  reporting  of  ACSEP 
evaluation  results.  Order  8100.7  provides  guidance  and  assigns  responsibility  for  the 
implementation  of  the  ACSEP  and  are  vital  tools  in  assurance  of  the  FAA's  mission  of 
continued  operational  safety.  The  program  assesses  the  compliance  of  production 
approval  holders  and  delegated  facilities  to  the  requirements  of  applicable  CFR  and 
FAA-approved  data,  including  compliance  to  the  procedures  established  to  meet  those 
requirements.  It  also  surveys  the  application  of  standardized  evaluation  criteria  not 
required  by  the  CFR  to  identify  national  issues  that  may  require  development  of  new  or 
revised  regulations,  policy,  and  guidance. 

Evaluation  criteria  for  the  production  approval  holders  are  further  divided  into  17  system 
elements  for  detailed  data  collection  and  reporting.  The  17  system  elements  are: 

•  Organization  and  Responsibility  •  Supplier  Control 

•  Design  Data  Control  •  Nonconforming  Material 

•  Software  Quality  Assurance  •  Material  Handling/Storage 

•  Manufacturing  Processes  •  Airworthiness  Determination 

•  Special  Manufacturing  Processes  •  FAA  Reporting  Requirements 

•  Statistical  Quality  Control  (SQC)  •  Internal  Audit 

•  Tool  and  Gauge  •  Global  Production 

•  Testing  •  Manufacturing  Maintenance  Facility 

•  Nondestructive  Inspection 
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These  system  elements  contain  criteria  that  assess  compliance  to  the  various  requirements 
of  the  CFR,  FAA-approved  data,  and  implementation  of  accepted  industry  practices.  In 
total  there  are  228  evaluation  criteria  in  the  manufacturing  portion  of  ACSEP.  However, 
the  number  of  evaluation  criteria  contained  in  these  system  elements  varies  and  is  not 
equally  proportioned  to  each  facility  type.  The  amount  of  variation  is  due  to  the  CFR 
requirements  and  industry  practices  for  the  different  facility  types.  The  17  system 
elements  vary  in  proportion  from  a  high  side  of  26  evaluation  criteria  or  12  percent  of  the 
total  for  Manufacturing  Processes  to  a  low  side  of  two  evaluation  criteria  or  1  percent  for 
Internal  Audit  (reference yigwre  A-1). 


Airworthiness 
Material  Determination 
Handling/Storage  4% 


Nonconforming 
Material 
5% 

Supplier  Control 
7% 


FAA  Reporting 
Requirements 

I 


Nondestructive 
Inspection  (NDI) 
8% 


Testing 

6% 


Tool  and  Gauge 
10% 


Internal  Audit  (2) 
^  1% 


Global  Production 
2% 

Manufacturing 
^Maintenance  Facility 
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Organization  & 
Responsibility 
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Software  Quality 
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3% 


Manufacturing 

Process 

12% 


Figure  A- 1.  —Evaluation  criteria  distribution  within  the  1 7  system  elements  of  ACSEP  for 

production  approval  holders. 
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Evaluation  criteria  for  delegated  facilities  are  divided  into  ten  system  elements.  The  ten 
system  elements  are: 


•  Organization  and  Responsibility 

•  Design  Data  Approval 

•  Testing 

•  Airworthiness  Certification 

•  Continued  Airworthiness 


Project  Management 
Design  Change  Approval 
Conformity  Inspection 
FAA  Notification 
Audit 


Similar  to  the  system  elements  for  production  approval  holders,  these  system  elements 
contain  criteria  that  assess  compliance  to  the  various  requirements  of  the  CFR, 
FAA-approved  data,  and  implementation  of  accepted  industry  practices.  In  total  there  are 
114  evaluation  criteria  in  the  delegated  facility  portion  of  ACSEP.  However,  the  number 
of  evaluation  criteria  contained  in  these  system  elements  varies.  The  amount  of  variation 
is  due  to  the  CFR  requirements  and  industry  practices.  The  10  system  elements  vary  in 
proportion  from  a  high  side  of  27  evaluation  criteria  or  23  percent  of  the  total  for  Project 
Management  to  a  low  side  of  4  evaluation  criteria  or  4  percent  for  Audit  and  FAA 
Notification  (reference  figure  A-2). 


Design  Change 
Approval 
6% 


Design  Data 
Approval 
11% 


Project  Management 
23% 


FAA  Notification 


Airworthiness 
Certification 
7% 


Organization  & 
Responsibility 
17% 


Continued 
Airworthiness 
12% 


Audit 

4% 


Figure  A-2.  — Evaluation  criteria  distribution  within  the  10  system  elements  of  ACSEP  for 

delegated facilities. 
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A3.  Evaluations  and  Evaluators 

The  ACSEP  utilizes  teams  of  FAA  engineering,  flight  test,  and  manufacturing  inspection 
personnel  to  evaluate  production  approval  holders  and  delegated  facilities.  Upon 
completion  of  each  ACSEP  evaluation,  the  team  leader  prepares  a  report  and  forwards  it 
to  the  Certificate  Management  Office  (Manufacturing  Inspection  Office  or  Aircraft 
Certification  Office,  as  applicable)  which  provides  it  to  the  Aviation  Safety  Inspector 
(ASI)  and/or  the  Assigned  Engineer  (AE)  responsible  for  the  evaluated  facility.  A  copy 
of  the  report  is  also  provided  to  AIR-200  for  entry  into  the  ACSEP  database.  The 
ACSEP  database  contains  administrative  information  on  facilities  evaluated,  status  of 
qualified  team  members  and  team  leaders,  responses  to  rating  criteria  contained  in  the 
evaluation  system  elements,  along  with  findings  and  observations  noted.  Additionally, 
the  ACSEP  Master  Schedule,  which  is  prepared  annually,  is  maintained  by  AIR-200 
together  with  the  directorate  coordinators.  The  scheduling  database  is  updated  and  posted 
to  a  service  wide  electronic  mail  bulletin  board  on  a  monthly  basis  ensuring  the  Aircraft 
Certification  Service  offices  are  kept  current  of  ACSEP  evaluation  cancellations,  date 
changes,  and  recent  additions. 

The  frequency  at  which  production  approval  holders  are  scheduled  for  evaluation  is 
determined  by  Resource  Targeting.  The  design  of  Resource  Targeting  began  in  1994 
with  the  following  objective:  use  a  systematic,  analytic  approach  to  focus  the  FAA’s 
limited  resources  on  evaluating  those  facilities  with  the  greatest  potential  safety  impact. 
The  main  way  this  objective  was  to  be  met  was  to  adjust  the  frequency  at  which  facilities 
would  be  evaluated.  Resomce  Targeting  uses  a  process  of  assessing  the  risks  and 
scheduling  those  facilities  with  the  greatest  perceived  risk  more  frequently  than  facilities 
with  less  perceived  risk.  Aimually,  each  approval  holder  is  assessed  with  21  safety 
factors  and  the  criticality  of  the  parts  they  manufacture.  The  21  safety  factors  and  part 
criticality  are  split  into  two  aggregate  factors:  system  strength  and  inherent  risk.  System 
strength  is  a  measure  of  how  capable  the  quality  system  is  of  ensuring  that  parts  will  be 
manufactured  according  to  FAA-approved  data.  Inherent  risk  measures  the  risk  that  a 
part  failure  would  have  on  continued  operational  safety.  The  collective  score  of  the  two 
aggregate-factors  determines  which  of  the  four  RT  groups  is  assigned  to  the  facility.  Its 
RT  group  determines  the  frequency  at  which  a  facility  is  evaluated: 

RT  group  I:  evaluated  every  16  to  24  months 

RT  group  II:  evaluated  every  24  to  36  months 

RT  group  III:  evaluated  every  32  to  48  months 

Delegated  facilities  are  scheduled  for  evduation  according  to  their  delegation:  DOA  and 
DAS  facilities  are  scheduled  every  24  months  and  SFAR-36  facilities  are  scheduled  for 
evaluation  every  36  months. 

At  the  conclusion  of  an  ACSEP  evaluation,  a  post-evaluation  conference  is  held  with  the 
evaluated  facility  management  and  any  issues,  findings,  and/or  observations  are 
reviewed.  The  ASI  and/or  AE  responsible  for  facility  surveillance  pursue  any  findings 
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that  require  formal  corrective  action.  The  ASI  and/or  AE  inform  the  facility  of  the 
findings  and  request  corrective  action  though  a  Letter  of  Investigation,  when  deemed 
appropriate. 

The  ACSEP  also  includes  a  Quality  Improvement  Program.  Data  from  the  evaluation 
feedback  reports  and  evaluation  reports  are  used  to  prompt  improvements  in  the  program. 
Continuous  improvement  teams  established  in  each  directorate  and  in  headquarters 
review  suggestions,  comments,  and  results  of  the  evaluations.  The  directorate  teams  act 
upon  improvements  that  can  be  implemented  locally;  improvements  that  affect  the 
national  program  are  referred  to  a  dedicated  National  Continuous  Improvement  Team 
(NCIT)  made  up  of  FAA  Aviation  Safety  Inspectors,  Aerospace  Engineers,  and  Flight 
Test  Pilots  representing  the  directorates  and  headquarters.  Managers  representing  the 
Aircraft  Certification  Management  Team  (ACMT),  Aircraft  Certification  Office 
Management  Team  (ACOMT),  and  Manufacturing  Inspection  Management  Team 
(MIMT)  are  also  members  of  the  National  Continuous  Improvement  Team  (NCIT).  After 
a  comprehensive  review  of  the  data,  the  NCIT  recommends  changes  or  clarification  to 
current  policy.  Recommended  changes  are  forwarded  to  the  Aircraft  Engineering 
Division  (AIR- 100)  or  the  Production  and  Airworthiness  Certification  Division 
(AIR-200)  for  further  review  and  possible  implementation. 

The  AIR  organization  is  responsible  for  conducting  evaluator  training.  This  is 
accomplished  in  association  with  the  FAA  Academy  with  AIR-200  providing  instructors. 
These  instructors  are  experienced  national  evaluation  team  leaders  who  bring  real  life 
experiences  into  the  classroom.  While  one  instructor  presents  the  course  materials,  the 
other  critiques  the  presentation/materials  and  notes  comments  from  students.  The 
critique  and  notes  are  reviewed  and  improvements  incorporated  facilitating  a  continuous 
improvement  process.  Additionally,  issues  found  in  the  field  are  also  integrated  into  the 
course  making  it  even  more  comprehensive  and  continuously  improving  it. 
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APPENDIX  B.  DEFINITIONS 

Approved  Production  Inspection  System  (APIS)  -  Federal  Aviation  Administration 

(FAA)  production  approval  issued  to  a  manufacturer  of  an  aircraft,  aircraft  engine, 
or  propeller  being  manufactured  under  a  type  certificate  only. 

Assigned  Engineer  -  An  FAA  engineer  to  whom  the  Aircraft  Certification  Office 

manager  has  assigned  responsibility  relating  to  ACSEP  evaluations  at  a  particular 
design  approval  facility. 

Compliance  -  for  the  purposes  of  this  report,  compliance  refers  to  a  facility’s  business 
practices  being  consistent  with  published  procedures  and/or  policies.  These 
procedures/policies  include:  internal  procedures/policies  not  requiring  FAA 
approval,  FAA-approved  data,  and  the  CFR. 

Compliance  Rate  -  the  proportion  of  facilities  whose  business  practices  were  formd  to  be 
in  compliance  With  published  procedures  and/or  policies  at  the  time  of  an  ACSEP 
evaluation.  These  procedures/policies  include:  internal  procedures/policies  not 
requiring  FAA  approval,  FAA-approved  data,  and  the  CFR. 

Criteria  -  the  basic  element  of  an  ACSEP  evaluation.  Criteria  are  used  to  plan  the  depth 
of  the  evaluation  and  to  document  the  results  of  the  evaluation  in  a  standardized 
manner.  The  criteria  are  grouped  into  systems  and  system  elements. 

Delegated  Facility  -  a  facility  undertaking  DO  A,  DAS,  or  SFAR-36  activity. 

Delegation  Option  Authorization  (DO A)  -  an  organization  or  facility  authorized  by  the 
FAA  to  accomplish  type,  production,  and  airworthiness  certification  of  certain 
products  as  specified  in  CFR  §  21.231(a). 

Designated  Alteration  Station  (DAS)  -  an  organization  or  facility  authorized  by  the  FAA 
to  issue  supplemental  type  certifications,  experimental  certificates,  and  amended 
standard  airworthiness  certificates  in  accordance  with  its  FAA-approved 
procedures  manual. 

Established  Industry  Practice  -  a  widely  followed  method  of  operating  that  achieves 
consistent  performance  of  specific  functions  (i.e.,  calibration  recall  system, 
internal  audit  system,  and  statistical  process  control). 

Facility  -  for  this  report,  any  production  approval  holder,  delegation,  or  priority  part 
supplier. 
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CFR-based  Observation  -  an  occurrence  of  FAA-approved  data  not  in  compliance  to  the 
Code  of  Federal  Regulations  (CFR). 

Federal  Aviation  Regulations  (FAR)  -  regulations  listed  in  Title  14  (Aeronautics  and 
Space)  of  the  CFR. 

Finding  -  systemic  noncompliance  to  the  CFR,  FAA-approved  data  (or  in  the  case  of 

supplier  facilities,  the  purchasing  instrument),  or  a  safety-related  noncompliance. 

Issue  —  An  inconsistency  between  the  actual  operating  practices  of  a  facility  and  the  CFR, 
FAA-approved  data,  or  the  facility’s  internal  procedures. 

Isolated  Observation  -  isolated  occurrence  of  noncompliance  to  the  CFR  or 
FAA-approved  data. 

Manufacturer's  Maintenance  Facility  (MMF)  -  defined  by  CFR  §  145.1(c)  as  a  repair 
station  certificate  with  a  limited  rating  issued  to  a  manufacturer  based  upon  the 
production  approval  it  holds  fi-om  the  FAA. 


National  Continuous  Improvement  Team  (NCIT)  -  a  dedicated  national  team  of  FAA 
aviation  safety  inspectors,  aerospace  engineers,  flight  test  pilots,  and  managers 
representing  the  directorates  and  divisions  chartered  to  review  the  ACSEP 
periodically  for  areas  of  improvement. 

Noncompliance  -  for  the  purposes  of  this  report,  noncompliance  refers  to  a  facility’s 

business  practices  being  inconsistent  with  published  procedures  and  policies  at  the 
time  of  the  ACSEP  evaluation.  These  procedures  and/or  policies  include: 
internal  procedures/policies  not  requiring  FAA  approval,  FAA-approved  data,  and 
the  CFR. 

Noncompliance  Rate  -  the  proportion  of  facilities  where  at  least  one  business  practice 
was  inconsistent  with  published  procedures  or  policies,  or  any  portion  thereof,  at 
the  time  of  the  ACSEP  evaluation.  These  procedures  and/or  policies  include: 
internal  procedures  not  requiring  FAA  approval,  FAA-approved  data,  and  the 
CFR. 

Nonobservance  -  a  failure  to  comply  with  self-imposed  procedures  that  are  related  to,  but 
not  required  by,  the  applicable  production  approval,  delegated  facility  approvi,  or 
quality  requirements  from  a  parent  manufacturing  maintenance  facility. 
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Parts  Manufacturer  Approval  (PMA)  -  an  FAA  production  and  design  approval  issued  to 
manufacturers  who  produce  replacement  or  modification  parts,  equipment, 
components,  materials,  part  processes  (replacement  and  modification,  and 
appliances). 

Principal  Inspector  (PI)  -  an  FAA  aviation  safety  inspector  who  has  been  assigned 

certificate  management  and/or  surveillance  responsibility  for  a  PAH,  associate 
facility,  or  priority  part  supplier. 

Priority  Part  Supplier  (PPS)  -  any  person  or  organization  (including  a  distributor)  that 
furnishes  priority  parts  (as  defined  in  Order  8120.2)  to  a  PAH. 

Production  Approval  Holder  (PAH)  -  the  holder  of  a  PC,  APIS,  PMA,  or  TSO 

authorization,  who  controls  the  design  and  quality  of  a  product  or  part  thereof. 

Production  Certificate  (PC)  -  an  FAA  production  approval  issued  to  a  manufacturer  of 
aircraft,  aircraft  engines,  or  propellers  that  has  had  its  Quality  Control  system 
examined  and  approved  by  the  FAA,  and  that  holds  one  or  more  of  the  following: 
a  current  type  certificate,  rights  to  the  benefits  of  a  type  certificate  under  a 
licensing  agreement,  or  a  supplemental  type  certificate. 

Production  Certificate  Extension  (PCEJQ  -  an  FAA-approved  extension  of  a  specific 
manufacturer's  PC  to  another  facility. 

Safety  Finding  -  safety-related  noncompliance  that  requires  immediate  action. 

Special  Federal  Aviation  Regulation  No.  36  (SFAR-36)  -  an  organization  or  facility 
authorized  by  the  FAA  to  approve  major  repairs  on  a  product  or  article  in 
accordance  vsdth  its  FAA-approved  procedures  manual. 

System  element  -  a  logical  grouping  of  several  criteria  into  functional  areas.  There  are  17 
system  elements  for  production  approval  holders  and  10  system  elements  for 
delegated  facilities. 

System  —  the  highest  level  of  grouping  for  the  ACSEP  criteria.  Systems  comprise  the 
individual  disciplines  imder  which  the  criteria  fall.  There  are  six  systems: 
Management,  Engineering,  Manufacturing,  Quality,  Service/Product  Support,  and 
Communication  with  the  FAA. 

Systemic  Issue  -  either  a  finding  or  a  systemic  observation. 

Systemic  Observation  -  systemic  nonobservance  to  other  than  FAA  requirements  or 
FAA-approved  data. 
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Technical  Standard  Order  (TSO)  authorization—  an  FAA  design  and  production  approval 
issued  to  a  manufacturer  for  an  article  which  has  been  found  to  meet  a  specific 
FAA  Technical  Standard  Order. 
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